One of the most interesting features of the Pushpin proxy is its ability to gateway between WebSocket clients and plain HTTP backend servers. In this article, we’ll demonstrate how to build a WebSocket service using Express as the HTTP backend behind Pushpin.

Wait, HTTP?

Both Pushpin and Node support WebSockets natively, so you might wonder why you’d want to use Pushpin’s HTTP-gatewaying feature between the two. The reason is that it enables stateless backend development. Often, WebSocket services send messages to clients in two cases:

  1. In reaction to an incoming message from the client (e.g. some kind of RPC).
  2. To relay a spontaneous “push” event generated by a backend service.

When building a WebSocket service using Pushpin, you’d almost always want to handle the second case by publishing messages to Pushpin out-of-band. In theory, this means that the backend server should only need to send messages over the proxied connection to the client if it’s responding to a message that the client had just sent. This may not be the case with all WebSocket services, but it likely applies to many of them, if not most.

The resulting architecture looks like this:

pushpin-express

The Express application only needs to be able to handle short-lived HTTP requests, and make outbound HTTP POST requests whenever there is data to “push”. The Express application does not need to maintain any long-lived, stateful connections.

By trading WebSockets for HTTP at the proxy->backend path, we gain some nice things:

  1. Simpler code. No stateful connections in the backend makes it easier to understand, maintain, and test.
  2. Load balancing requests associated with a single client connection to a set of backends.
  3. Hot reload. Restarting the backend doesn’t disconnect clients.
  4. Less sockets between proxy and backend. If you have a multi-tiered architecture and a million client connections, it’s preferable to avoid having a million sockets at each tier.
  5. Scaling to handle more listening connections is straightforward: just add more Pushpin instances.

WebSocket-over-HTTP protocol

When Pushpin gateways a WebSocket connection over HTTP, events from the WebSocket connection are encoded into HTTP requests and sent to the backend server. If the server wishes to send events back to the WebSocket client, it encodes events in the HTTP responses.

Below are some example exchanges. The format is inspired by HTTP chunked encoding. Note that the characters \r\n represent a two-byte carriage return and newline sequence. Linebreaks are also inserted for readability.

Proxy tells the backend server about a new connection:

POST /target HTTP/1.1
Connection-Id: b5ea0e11
Content-Type: application/websocket-events

OPEN\r\n

Backend server accepts connection:

HTTP/1.1 200 OK
Content-Type: application/websocket-events

OPEN\r\n

Proxy relays message from client:

POST /target HTTP/1.1
Connection-Id: b5ea0e11
Content-Type: application/websocket-events

TEXT 5\r\n
hello\r\n

Backend server responds with two messages:

HTTP/1.1 200 OK
Content-Type: application/websocket-events

TEXT 5\r\n
world\r\n
TEXT 1B\r\n
here is another nice message\r\n

For details, see the spec. Note that it’s not necessary to understand the protocol since the express-grip library (discussed below) takes care of it for you.

express-grip and mock socket object

The express-grip library provides a socket-like object called WebSocketContext that handles the event marshalling over HTTP. The object contains methods like accept(), send(), recv(), etc. What’s interesting is that these methods don’t operate directly on a real WebSocket. When recv() is called, it simply iterates over the events received in the current HTTP request. When send() is called, events are temporarily enqueued and a middleware serializes them at the end into the HTTP response. WebSocketContext objects are not long-lived, and a fresh one is created for each handler invocation and destroyed afterwards.

Code

We’ll make a simple echo & broadcast service. There will be two endpoints:

  • /websocket - Endpoint for WebSocket clients to connect to. Any received messages will be echoed back to the client.
  • /broadcast - HTTP endpoint. Accepts POST requests, and the request body is sent as a string to all connected WebSocket clients.

Pushpin will be configured with the route:

* localhost:3000,over_http

This means to route all incoming traffic to port 3000 on the local machine, with WebSocket connections gatewayed as HTTP.

Below is the full code for the Express backend. Afterwards, we’ll go over the important parts of it.

var express = require('express');
var bodyParser = require('body-parser');
var grip = require('grip');
var expressGrip = require('express-grip');

expressGrip.configure({
    gripProxies: [
        // pushpin config
        {
            'control_uri': 'http://localhost:5561';
        }
    ]
});

var app = express();

// Add the pre-handler middleware to the front of the stack
app.use(expressGrip.preHandlerGripMiddleware);

app.all('/websocket', function(req, res, next) {
    // Reject non-WebSocket requests
    if (!expressGrip.verifyIsWebSocket(res, next)) {
        return;
    }

    var ws = expressGrip.getWsContext(res);

    // If this is a new connection, accept it and subscribe it to a channel
    if (ws.isOpening()) {
        ws.accept();
        ws.subscribe('all');
    }

    while (ws.canRecv()) {
        var message = ws.recv();

        // If return value is null then connection is closed
        if (message == null) {
            ws.close();
            break;
        }

        // Echo the message
        ws.send(message);
    }

    // next() must be called for the post-handler middleware to execute
    next();
});

app.post('/broadcast', bodyParser.text({
    type: '*/*'
}), function(req, res, next) {
    // Publish data to all clients that are connected to the echo endpoint
    var data = req.body;
    expressGrip.publish('all', new grip.WebSocketMessageFormat(data));
    res.send('Ok\n');

    // next() must be called for the post-handler middleware to execute
    next();
});

// Add the post-handler middleware to the back of the stack
app.use(expressGrip.postHandlerGripMiddleware);

var server = app.listen(3000, function () {
  var host = server.address().address;
  var port = server.address().port;
  console.log('Example app listening at http://%s:%s', host, port);
});

The first thing you’ll notice is Pushpin configuration:

expressGrip.configure({
    gripProxies: [
        // pushpin config
        {
            'control_uri': 'http://localhost:5561';
        }
    ]
});

This tells the express-grip library where published data should be sent to. We’re using Pushpin’s default control port on the local machine.

We also set up some middleware. There are two middleware handlers. One that goes before everything and one that goes after:

// Add the pre-handler middleware to the front of the stack
app.use(expressGrip.preHandlerGripMiddleware);

... other handlers ...

// Add the post-handler middleware to the back of the stack
app.use(expressGrip.postHandlerGripMiddleware);

With that boilerplate out of the way, let’s look at the actual handlers. In the /websocket handler, we accept all incoming connections and immediately subscribe them to a channel called all:

// If this is a new connection, accept it and subscribe it to a channel
if (ws.isOpening()) {
    ws.accept();
    ws.subscribe('all');
}

If the HTTP request contained a WebSocket OPEN event, then isOpening() will return true. Calling accept() simply sets an internal flag. When the handler finishes, the postHandlerGripMiddleware will see that this flag was set and include an OPEN event in the HTTP response that it sends back to the proxy. The subscribe() call is a shortcut for sending a GRIP subscribe control message as a TEXT event.

Then we process any incoming messages in a loop. If the HTTP request contained a CLOSE event, then recv() will return null. Similar to accept(), calling close() sets an internal flag telling the middleware to include a CLOSE event on the way out.

while (ws.canRecv()) {
    var message = ws.recv();

    // If return value is null then connection is closed
    if (message == null) {
        ws.close();
        break;
    }

    // Echo the message
    ws.send(message);
}

Lastly, in the /broadcast handler we publish a WebSocket message to the all channel, which every connection is subscribed to:

var data = req.body;
expressGrip.publish('all', new grip.WebSocketMessageFormat(data));

As with the /websocket handler, this handler is also stateless. The publish() call is fire-and-forget.

That’s all there is to it! The backend code looks like typical WebSocket handling logic, except it’s completely faked and stateless.